Autonomous vehicles, such as vehicles that do not require a human driver, can be used to aid in the transport of passengers or items from one location to another. Such vehicles may operate in a fully autonomous mode where passengers may provide some initial input, such as a pick up or destination location, and the vehicle maneuvers itself to that location. While doing so, safety of the passengers and the vehicle is an important consideration. Accordingly, often these vehicles have fallback systems which essentially cause the vehicle to apply the brakes as hard and as quickly as possible in an emergency.